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Abstract 

This study investigated the effects of synthetic vision system (SVS) concepts and 
advanced flight controls on the performance of pilots flying a light, single-engine general- 
aviation airplane. We evaluated the effects and interactions of two levels of terrain portrayal, 
guidance symbology, and flight control response type on pilot performance during the 
conduct of a relatively complex instrument approach procedure. The terrain and guidance 
presentations were evaluated as elements of an integrated primary flight display system. The 
approach procedure used in the study included a steeply descending, curved segment as might 
be encountered in emerging, required navigation performance (RNP) based procedures. Pilot 
performance measures consisted of flight technical performance, perceived workload, 
perceived situational awareness and subjective preference. The results revealed that an 
elevation based generic terrain portrayal significantly improved perceived situation 
awareness without adversely affecting flight technical performance or workload. Other 
factors (pilot instrument rating, control response type, and guidance symbology) were not 
found to significantly affect the performance measures. 
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Nomenclature 

ANOVA = analysis of variance 

BSBG = blue sky brown ground (terrain portrayal concept) 
df = degrees of freedom for analysis of variance 

EBG = elevation based generic (terrain portrayal concept) 

FBW = fly-by-wire 

FTP = flight technical performance 

F-value = ratio of the model mean square to the error mean square 

GSC = guidance symbology concept 

HITS = highway in the sky (guidance symbology concept) 

IFR = instrument flight rules 

IV = independent variable 

p = p-value (ANOVA significance level) 

PFD = primary flight display 

PRFD = pitch roll flight director (guidance symbology concept) 

RMSE = root mean squared error 

RNP = required navigation performance 

SA = situation awareness 

SART = Situation Awareness Rating Technique 

SD-HDD = Symbology Development for Head-Down Displays 

SVS = Synthetic Vision System 

TEX = Task Load Index 

TPC = terrain portrayal concept 

TWS = time within standard 

V-CAS = velocity vector augmentation system 
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Introduction 

Single-pilot operations in low-visibility conditions can be extremely challenging and 
less forgiving of human-error than multi-pilot operations. On average, the workload is higher 
and without the ability to cross-check decisions and actions with an independent crew 
member, slips and lapses have an increased risk of propagating into hazardous situations. At 
the same time, many single-pilot operations are conducted by pilots with less training, total 
experience, recent experience, and oversight than pilots typically conducting multi-pilot, 
commercial transport operations. Finally, full access to the future national airspace is likely 
to require high flight-technical performance while performing more complex procedures such 
as curved, radius-to-fix legs, and the ability to self-separate during some flight phases (JPDO, 
2007). Meeting these future requirements may be particularly challenging for single-pilots 
and may, more than crewed-operations, depend on the careful integration of advanced 
technologies. The demands of single-pilot operations put increased emphasis on technologies 
that, beyond performing their intended function, are easily learned and remembered, 
minimize adverse workload additions or peaks, and support robust error resistance and/or 
recovery. Two technologies that may be particularly beneficial in this context are synthetic 
vision and highly-augmented, manual flight controls. 

Synthetic Vision Systems (SVS) are designed to improve pilot performance by 
enhancing situation awareness (SA) and control precision without increasing mental and 
physical workload significantly. SVS typically 
refers to a primary flight display (PFD) that, in 
addition to the traditional control, performance, and 
navigation indicators, includes an egocentric, 
perspective rendering of the external environment 

Fieure 1 - Examnle of a SVS based PFD 
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(e.g. terrain, obstructions, and cultural features) as shown in figure 1. This rendering often 
includes a visualization of the desired flight path, usually in the form of a pathway or 
“highway in the sky” (HITS, Arthur, Prinzel, Kramer, Parrish, & Bailey, 2003). A HITS 
provides easily interpreted visual cues indicating the position and orientation of the aircraft 
relative to the desired flight path. To further enhance the usefulness of the HITS, a flight 
path marker (FPM) symbol showing the actual or near future (i.e. quickened or predicted) 
direction of travel of the aircraft is typically displayed as well. The expectation is that the 
naturalistically presented information of an SVS can be assimilated more rapidly and robustly 
than conventional presentations. Previous studies such as Glaab and Takallu (2002), Uenking 
and Hughes (2002), Comstock, Glaab, Prinzel, and Elliott (2001), and Hughes and Takallu 
(2002), support this expectation. 

A potential concern regarding SVS is that much of the terrain and pathway information 
is not essential for immediate control and guidance of the flight and can be visually complex 
and cognitively compelling, possibly interfering with the perception of other critical 
information (e.g. Wickens, Alexander, Horrey, Nunes, & Hardy, 2004). Wong, Takallu, 
Hughes, Bartolone, and Glaab (2004) conducted a simulation experiment to partially 
investigate this potential. They compared flight technical performance (FTP), workload, and 
situation awareness for a range of SVS symbology concepts ranging from relatively simple 
terrain and guidance presentations to potentially more informative but also more complex 
presentations. Their study found no interaction between terrain portrayal concepts and FTP 
and no significant interaction between guidance symbology and terrain portrayal complexity. 
These results suggest that designers can independently choose a preferred terrain portrayal 
and guidance symbology concept without excessive concern regarding adverse interactions. 
Our investigation examines key findings from Wong, et al. (2004) by evaluating a sub-set of 
their display concepts in a flight environment. 
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While SVS by itself has the potential to enhance FTP and pilot awareness, the 
underlying task of flying the airplane remains essentially unchanged. For a typical, 
unaugmented airplane, the pilot must monitor the primary flight instruments nearly full-time, 
particularly when operating in turbulence. Pennington (1979) reported that proficient pilots 
using conventional, electro-mechanical flight instruments allocate approximately 70-80% of 
their visual attention to monitoring the attitude indicator and directional gyro. This high 
allocation severely limits the time available for other important cockpit tasks. While large- 
format attitude and SVS displays should reduce this allocation, the basic flight characteristics 
of an aircraft require constant, high-frequency attention and brief inattention can result in an 
unusual attitude and possible loss of control (Newman & Greeley, 2001). 

Current autopilots offer a means of addressing this concern but create additional 
complexity and potential hazards by introducing multiple, dissimilar modes of control, and in 
more complex systems, potentially confusing temporal shifts between command inputs and 
the response of the airplane. In addition, autopilots encourage detachment from the basic 
“aviate” task by eliminating the pilot’s physical involvement (Billings, 1997). While the 
pilot is expected to monitor the situation, there is generally no immediate feedback or 
consequence if this responsibility is not diligently performed. 

An alternative to current autopilot systems is to integrate active control elements 
directly into the manual control system such that the short and long-term responses of the 
aircraft follow appropriate performance indices. A “Fly-by-wire” (FBW) control system in 
which the pilot’s inceptors, for example the control yoke, issue commands to a flight control 
computer rather than being linked directly to the control effectors underlies, perhaps, the 
definitive mechanization of such an approach. Although currently considered too expensive 
and high-risk for light aircraft, FBW technology has recently migrated from military and 
large commercial aircraft to lower cost commuter and business jets and technologically 
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similar “drive-by-wire” systems are also beginning to appear on production automobiles. 

These trends suggest that FBW may become economical for small aircraft in the foreseeable 
future. 

The application of FBW opens up a range of design options as to what the pilot 
commands through the control inceptors and how the aircraft responds to these commands 
and other factors of flight such as staying within the operating envelope. Depending on the 
goals of the design and practical constraints in its realization, for example, consideration of 
failure effects, FBW offers potential performance, training, workload, and safety benefits. 

By creating a direct, proportional relationship between operational parameters of interest (e.g., 
vertical speed, turn rate, and airspeed) and the airplane’s response to the pilot’s control 
inceptors, the effort to learn and preserve low-level perceptual-motor skills can be reduced, as 
can operational workload. Automatic disturbance rejection may also reduce workload and 
improve performance when the pilot’s attention is diverted from the immediate control task. 
Finally, integrated envelope protection features may improve safety by preventing 
unintentional departures from the design flight envelope and by simplifying the piloting 
technique needed to achieve and maintain maximum performance (Rogers, 1999). 

The design space of potential FBW realizations is large and relatively unexplored 
outside the perspective of traditional flying qualities for skilled pilots with undivided 
attention on a control task. The design of the system in this study was directed toward 
enabling training, workload, performance, and safety benefits, particularly for low-time and 
ab-initio student pilots. Since there are few detailed guidelines for achieving these objectives, 
the specific system implementation used in this study should be considered exploratory and 
not necessarily the optimum design relative to these goals. Extensive research would be 
required to propose any such recommendations and is far beyond the scope of this study. It 
should also be noted that while the system used in the study was developed with the goal of 
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demonstrating the full range of benefits, this study focuses on assessing workload and 
performance impacts during manual control. 

The research FBW system provided the evaluation pilots with direct control over the 
velocity vector of the aircraft and can be classified as a velocity -vector command 
augmentation system (V-CAS). The longitudinal position of a 2-axis side-stick commanded 
the vertical, air-mass referenced, flight-path angle. Lateral side-stick position commanded 
bank angle (effectively turn-rate and turn radius at constant airspeed). A separate, single-axis 
lever commanded airspeed (figure 2). It should be recognized that these response 
characteristics are quite different from an unaugmented airplane. For example, when the 
pilot applies no force to the stick, allowing it to 
return to neutral, the bank angle and flight-path 
angle commands are zero and the aircraft promptly 
returns to and maintains straight and level flight. 

In comparison, neutralizing the inputs of an 
unaugmented airplane simply returns the control 

Fisure 2 - Control incentors for FBW svstem 

effectors to a neutral position, nulling their respective control moments and nothing can be 
directly inferred about the attitude or trajectory. The current implementation was developed 
for ease of learning by ab initio pilots with no flight experience but highly experienced 
operating ground vehicles in which inceptor inputs (e.g. steering wheel deflection) typically 
correspond to the rate of change of the trajectory (e.g. turn rate). As would be expected, the 
unconventional response characteristics introduced transition issues for the experienced pilots 
used in this study. The transition issues were recognized during the development of the 
system but considered acceptable based on earlier research (e.g. Bergman, 1976). 

In simulation (Stewart, 1994; Lam, Mulder, van Paassen, & Mulder, 2006) and limited 
flight experiments (Bergman, 1976), similar V-CAS implementations show improved FTP 
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while reducing workload. Stewart also demonstrated that minimal training is needed to use a 
properly functioning V-CAS system (i.e. effects of potential failures have not been 
investigated). This study examines the general findings from these earlier studies in a flight 
environment and in the context of emerging airspace procedures incorporating curved flight 
segments. In addition, this study investigated potential interactions between the SVS and V- 
CAS concepts. Previous research (e.g. Stewart, 1994 and Lam, et al., 2006) suggests that the 
control-display concurrence of a path-based HITS and the path-based control provided by the 
V-CAS is particularly beneficial. 

Since the motivating context of this evaluation is small-aircraft operations, the flight 
task scenario and evaluation pilot pool were selected accordingly. The flight task, explained 
in detail later, consisted of a challenging, multi-segment approach procedure intended to be 
representative of what might be implemented in the future using required navigation 
performance (RNP) concepts. Such a procedure might be needed to support access to a 
terrain challenged airport or an urban airport with demanding noise and/or obstacle clearance 
concerns. The evaluation pilot pool consisted of current, licensed pilots and included both 
instrument rated and non-instrumented rated subjects. The investigators initially planned on 
including flight-naive subjects, but time limitations prevented this participation. 
Summarizing, the specific objectives of the study were two fold: 

1. Evaluate the benefits and issues of SVS displays and a V-CAS for a single pilot 
conducting advanced airspace procedures by providing comparative results of SVS 
and V-CAS versus conventional interfaces. 

2. Examine key findings from a previous simulation experiment (Wong, et al., 2004) in 
flight. Specifically, investigate the interactions between Guidance Symbology 
Concepts (GSC) and Terrain Portrayal Concepts (TPC) as part of a PFD. 
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Method 

Based on the objectives of the study, the following hypotheses were tested: 

Hypothesis 1. SVS/HITS significantly improves the flight performance of both IFR 
and non-IFR pilots. 

Hypothesis 2\ V-CAS has a significant improvement on both IFR and non-IFR pilots’ 
flight performance. 

Hypothesis 3: With SVS/HITS and V-CAS, non-IFR pilots can achieve the flight 
performance of an experienced IFR pilot with conventional controls. 

Hypothesis 4\ FTP will not be significantly affected by the 2 terrain portrayal concepts 
of the SVS display for both IFR and non-IFR pilots. 

Hypothesis 5: Pilots will have a significant subjective preference for elevation based 
generic (EBG) over blue sky, brown ground (BSBG) terrain portrayal. 

Due to limitations on the availability of suitable test subjects when the flights were 
conducted, the final mix of evaluation pilots was 8 Non IFR pilots and 4 IFR rated pilots. 
Also, while not a requirement for participation, all the evaluation pilots were male. 

To test these hypotheses with the small subject pool, a mixed factorial design was 
conducted on the following variables: 

Independent Variables (IVs): There were four independent variables with each variable 
having two levels: 

IV- 1: Terrain portrayal concepts [1. Blue sky, brown ground (BSBG); 2. Elevation based 
generic (EBG)] 

IV-2: Guidance and position awareness symbology concepts [1. Pitch / roll flight director 
(PRFD); 2. The preferred HITS symbology from Wong, et al., 2004 (e.g. the “NASA 
Ghost format”)] 

IV-3: Control system response types [1. Conventional aircraft controls; 2. V-CAS] 
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IV-4: Pilot Rating [1. Non IFR pilots; 2. IFR pilots] 

Among these variables, IV- 1, IV-2, IV-3 are within-subject variables and IV-4 is the 
only between-subject variable. There are 8 treatments for each pilot skill type: 4 symbology 
and terrain portrayal combinations x 2 the two control system types. Figures 3 and 4 
illustrate the 4 different combinations of the symbology and terrain portrayal types. Since 
changing the control system configuration took approximately one day to perform, it was 
necessary to evaluate one control system configuration at a time and keep this configuration 
until all other necessary runs had been made. For this reason, the control system presentation 
was counterbalanced with half the participants flying first with the V-CAS and the other half 
flying first with the conventional controls. Treatment presentation was randomized at a 
group level. This design should prevent practice effects from masquerading as treatment 
effects. That said, any practice effects will still reduce the sensitivity of the experiment. 



Figure 3. Terrain portrayal. Left: Blue Sky Brown Ground (BSBG)/NASA Tunnel 
(HITS); Right: Elevation Based Generic Terrain/ NASA Tunnel (EBG/HITS) 
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Figure 4. Illustration of symbology. Left: Blue Sky Brown Ground/ Pitch Roll Flight 
Director (BSBG/PRFD); Right: Elevation Based Generic Terrain / Pitch Roll Flight Director 


(EBG/PRFD) 


Apparatus 

Test airplane was a modified 1978 Model 
F33C Bonanza, S/N CJ-144. Figure 5 provides a 
picture of the modified cockpit. The right side of 
the cockpit served as a safety pilot’s station and 
retained conventional, certified instruments, 
avionics, and controls. During flight operations, a 
safety pilot was the legal, pilot in command. The left side of the cockpit was modified to 
serve as a flexible, evaluation pilot’s station and was equipped with reconfigurable controls 
and displays. When evaluating the V-CAS, a side-stick control inceptor and airspeed 
command lever were installed. In response to the pilot’s inputs, these devices generated 
signals sent to a flight control computer in the rear of the aircraft. For conventional control 
evaluations, a standard dual-yoke control column was installed. The evaluation pilot’s station 
was configured with two high-brightness 8x10 inches liquid crystal displays with 
resolutions of 1024 x 768 pixels. The left display was used to display the PFD concepts 
while the right display was used to present a complimentary navigation display. This 
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navigation display provided a real-time, planform view of the approach procedure and terrain 
in a track-up, exocentric format. Two personal computers mounted in the rear of the aircraft 
drove the two displays. Both of these computers received position and state information from 
an air-data, attitude and heading reference system installed in the aircraft. The computer 
driving the PFD also functioned as the data acquisition system, recording the performance 
parameters used later in the paper. 

Procedures 

Training 

All the evaluation pilots participated in the previously conducted “Symbology 
Development for Head-Down Display” (SD-HDD) simulation experiments described by 
Wong, et al. (2004) and Takallu, et al. (2004) and received extensive training and practice 
with the terrain portrayal and guidance symbology concepts. See Takallu, et al. (2004) for 
more details on this training. Since control system response type was not a factor in the SD- 
HDD experiment, subjects did not have prior training on the V-CAS. Prior to the conduct of 
the current flight experiment, subject pilots received a refresher briefing on the terrain 
portrayal and guidance symbology concepts as well as an introduction to the V-CAS. 

Subjects were also provided with sufficient flight time in the aircraft to become comfortable 
with the symbology and control concepts prior to the collection of relevant data. 

Evaluation tasks 

The Juneau approach procedure from the SD-HDD experiment was used in this study 
as the flight task. As Figure 6 illustrates, the approach consists of four different segments 
presenting the pilot with differing levels of difficulty. The procedure begins with an easy 
segment (straight and level) followed by a transition to a straight segment with a three degree 
descent angle typical of current procedures. This is followed by constant radius, curved 
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segment with a steep descent angle (6 degrees). At the approach speed of the test aircraft (90 
knots) the radius of curvature corresponds to a nominal bank angle of 10 degrees. The 
approach ends with a final straight segment having a 4 degree glide slope angle. 



Flight operations 

Flights originated from Beech Field in Wichita Kansas and the evaluation approaches 
were conducted nearby, in an area selected to avoid other air traffic. The research system 
allowed the measured position of the aircraft to be biased such that from the perspective of 
the SVS displays, the aircraft appeared to be operating in the Juneau, Alaska, area. 

Each evaluation pilot flew two separate flights corresponding to the two control system 
configurations. Individual flights lasted approximately 60 minutes including transit time to 
and from the test area. Within a flight, the pilots experienced the 4 display treatments with 
the order of presentation being randomized. During the conduct of the evaluation scenarios, 
subject pilots wore a view-limiting hood that prevented use of outside visual cues. At the 
completion of an evaluation approach, the safety pilot would take back control of the aircraft 
and set up for the next approach. During this period, the evaluation pilot completed the 
workload and SA questionnaires described in the next section. 
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Variables and Measurements (Dependent Variables) 

FTP measures how a pilot performs in terms his flight path and airspeed control 
relative to the desired path and airspeed. FTP was measured using two types of metrics: root 
mean square error (RMSE) and time within standard (TWS). Since RMSE is the square root 
of the averaged mean square of the deviation from standard (i.e., airspeed, vertical and lateral 
deviation) it should be noted that it cannot be negative. Given this lower bound of zero, the 
distribution of RMSE is not likely to be normal, which is a basic requirement for most 
inferential statistic procedures. To address this issue, RMSE data were transformed using the 
natural logarithm function, and this will usually result in a normal distribution of RMSE data. 
TWS is computed as the percentage of time during an approach during which the pilot 
remained within the specified performance tolerances. The tolerances used were airspeed 
within ±10 knots of 90 kts, lateral deviation <200 feet, and vertical deviation <150 feet. 

Subject pilots completed questionnaires after each evaluation approach (the run 
questionnaire), flight (the block questionnaire), and at the conclusion of a subject’s 
participation (the exit questionnaire). The run questionnaire focused on subjective evaluation 
of workload and SA using the NASA Task Load Index (TLX) and Situation Awareness 
Rating Technique (SART) respectively. Using the TLX, evaluators rated seven different 
workload factors: mental demand, physical demand, temporal demand, effort, performance, 
frustration, and stress level. Evaluators indicated their ranking by making a pencil mark 
along a continuum (indicated by a horizontal line segment) running from “low” to “high” for 
each factor. During the data analysis process, the placement of the marks was recoded into a 
quantitative value between 0-100 based on its position along the line segment. The raw 
rating of the “performance” factor was also subtracted from 100, so that like the other factors, 
a lower rating value generally indicates improved performance. This consistency allowed a 
composite workload rating to be obtained from an average of the factor ratings. Similar to 
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the TLX questionnaire, the SART survey assesses different aspects of pilot situation 
awareness, including demand on attention resource, supply of attention resource, level of 
aircraft situation awareness, level of terrain awareness, and level of guidance information 
awareness. As with the TLX, the SART was administered with paper and pencil with 
evaluators placing a mark for each factor along a continuum between low and high. Again, 
as part of the data analysis process, the placement of the marks was recoded into quantitative 
values between 0-100. For the SART responses, higher values typically indicate improved 
perceived awareness. Again, the average of the factor ratings was used to obtain a composite 
value. 

The block questionnaire was administered after each flight to obtain a participant’s 
subjective feedback regarding preferences towards different terrain portrayal and guidance 
symbology under each control type. After a subject completed both flights, he completed a 
questionnaire providing subjective feedback comparing the two control types. 

Results 

The twelve participants were scheduled to complete two flights for the study. Although 
all 12 participants completed all the planned evaluation approaches, there were some runs 
that had technical glitches and had to be repeated. Also, a data acquisition and recording 
system glitch resulted in the loss of lateral deviation data for one subject, so that that 
person’s data was dropped from the data analysis. In order to identify the significant factors, 
in depth statistical analyses were carried out by performing repeated measures ANOVA on 
each of the flight performance metrics. A significance level of 5% was used to identify the 
significant factors. The results are presented in the following sections. 
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Lateral Path Deviation 

The ANOVA results for lateral flight path deviation are presented in Table 1 . The 
results indicate that none of the main factors significantly influence lateral flight path 
deviation. Thus, based in this lateral path control, hypotheses 1, 2 and 3 are rejected and 
hypothesis 4 is accepted. 

Two interaction factors, however, were found to be significant. A further look at these 
interactions (Figure 7) shows an interaction between the control type and instrument rating. 


Table 1 - Results of ANOVA on horizontal deviation RMSE 


Source 

Design type 

df 

Sum of Square 

Mean Square 

F-value 

Significance 

p-value 

Pilot Rating 

Between 

subjects 

1 

.417 

All 

.235 

.639 

Control Type 

Within 

Subjects 

1 

1.046 

1.046 

1.184 

.305 

Terrain Portrayal 

Within 

Subjects 

1 

1.251 

1.251 

1.486 

.254 

Guidance 

Symbology 

Within 

subjects 

1 

5.418 

5.418 

3.932 

.079 

Control x Rating 

Within 

subjects 

1 

4.686 

4.686 

5.302 

.047* 

Control x 
guidance 
symbology x 
rating 

Within 

subjects 

1 

4.895 

4.895 

5.491 

.044* 


* Significant factor 



Figure 7 -- Mean horizontal RMSE for pilot 
rating and control type 


That is to say, for instrument rated (IFR) pilots, their performance on maintaining 


lateral position using V-CAS was degraded compared to conventional control, while for non- 
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IFR pilots, V-CAS improved this performance. This effect can be further investigated by 
using a 3-way interaction plot. The results show that with guidance symbology 1 (baseline 
PRFD), the interaction between control type and pilot rating is positive. For the PRFD 
guidance, the non-IFR pilots had larger lateral deviations than IFR rated pilots. For this 
symbology, both groups had increased deviation with the V-CAS. With the HITS symbology, 
the interaction between control type and pilot rating become negative, that is compared to the 
IFR rated pilots, the non-IFR rated pilots had higher deviation with the conventional control 
and lower deviation with the V-CAS. The above finding (negative interaction effect) results 
in rejecting hypothesis 2, which states that V-CAS will improve flight performance for both 
pilot groups. The results also suggest that the effect of V-CAS (positive or negative) on pilot 
performance relies on other factors, such as the guidance symbology. 


Vertical Path Deviation 

The ANOVA results for vertical flight path deviation are presented in Table 2: 


Table 2 - Results of ANOVA on vertical deviation RMSE 


Source 

Design type 

df 

Sum of Square 

Mean Square 

F-value 

Significance 

p-value 

Pilot Rating 

Between 

subjects 

1 

2.041 

2.041 

2.192 

.173 

Control Type 

Within 

subjects 

1 

.297 

.297 

.940 

.383 

Terrain Portrayal 

Within 

subjects 

1 

.081 

.081 

.154 

.703 

Guidance 

Symbology 

Within 

subjects 

1 

5.597 

5.597 

22.580 

.001* 


* Significant factor 


The only factor found significant for vertical deviation RMSE is the guidance 
symbology. The HITS symbology resulted in less vertical error than the baseline PRFD 
symbology across all other conditions. So in terms of vertical flight errors, hypothesis 1 can 
be accepted, as can hypothesis 4. Hypotheses 2 and 3 are rejected. It can also be noted that 
overall, across all conditions, IFR pilots performed better than non-IFR pilots in terms of 
controlling the glide slope of the aircraft, although not at a significant level. 
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Airspeed regulation 

The repeated measure ANOVA found no significant effect of any of the factors on the 
pilots’ ability to maintain airspeed at the 90 knots reference value. As a component of FTP, 
this analysis partially tests hypotheses 1, 2, 3, and 4. Since none of the factors were found to 
be significant, based on airspeed control, hypotheses 1, 2 and 3 are rejected and hypothesis 4 
is accepted. 

Time Within Standard (TWS) Metric 

A pilot was considered within standard if he was flying at the airspeed 90 ± 10 knots, 

with horizontal deviation less than 200 feet, and vertical deviation less than 150 feet. There 

were no significant factors or interactions identified for the TWS metric. Thus hypotheses 1, 

2 and 3 are rejected and hypothesis 4 is 

supported. A further look at the mean TMS as 

a function of pilot rating and control response 

type (Figure 8) illustrates that overall, IFR 

pilots had better performance than non-IFR 

pilots in terms of TWS, although as already 

Figure 8 - Mean TWS for pilot rating and mentioned, the difference did not reach 

control type 

significant levels in this study. 

Perceived Workload Assessment 

A repeated measure ANOVA was carried out, with the average workload scores of all 
seven domains as the dependent measure. ANOVA results reveal that the between-subject 
factor (pilot rating) was not a significant factor for TLX workload scores (p value of 0.458) 
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and none of the wi thin-subject factors were found significant (for control type, p=0.097; for 
terrain portrayal, p = 0.695; and for guidance symbology, p=0. 101). The only factor found 
significant is the 3-way interaction among control, terrain portrayal, and pilot rating (with p = 
0.01 1). A further investigation on this interaction found that under PRFD symbology, IFR 
pilots have higher workload scores for EBG while non-IFR pilots have higher workload 
scores for BSBG. With the HITS symbology, this interaction is reversed, that is, IFR pilots 
have a higher workload score for BSBG while non-IFR pilots have a higher score for EBG. 
The overall lowest score is IFR-BSBG-PRFD. 

Perceived Situation Awareness Assessment 

A repeated measure ANOVA analysis was carried out on the average SA score. Of all 
the factors, terrain portrayal was found as the only significant factor to affect SA (p < 0.001). 
The result strongly implies that under EBG terrain, pilots are more likely to maintain higher 
levels of situation awareness than BSBG. 

Block and Exit Questionnaire Results 

As described earlier, a block questionnaire was administered after each flight to obtain 
the participant’s subjective feedback regarding preferences towards different terrain portrayal 
and guidance symbology under each control type. And, after both flights were completed, 


each participant was asked to complete an exit questionnaire eliciting subjective feedback on 
the two control types. Table 3 presents the results of the exit questionnaires. 


Pilot rating 

Prefer Conventional 
(Percentage) 

Prefer V-CAS 
(percentage) 

Total 

Instrument Rating (IFR) 

2 (66%) 

1(33%) 

3 (100%) 

Non-instrument 

Rating(Non-IFR) 

6(75%) 

2(25%) 

8(100%) 


Table 3 - Summary of subjective preference on control types 
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From the exit survey results, it can be seen that regardless of pilot rating (IFR or Non- 
IFR), conventional control was preferred over the V-CAS control implementation. The 
reasons given by participants for this preference were often related to their familiarity with 
conventional control. Since all the pilots were trained using conventional flight control, this 
result is not unexpected given their comparatively limited exposure to the V-CAS. In general, 
the increased familiarity resulted in a greater sense of being in control. That said, the V-CAS 
does separate the pilot from the instantaneous activity of the control effectors, so in a real 
sense, there is a reduction in the pilot’s authority over the lowest-level actions of the airplane. 
For instance, many pilots commented, with some concern, about interactions between flight 
path commands (i.e. longitudinal stick inputs) and changes to the engines power-setting. 
Representative pilot comments by those favoring conventional control include the following: 
“I was in control with the conventional. Putting the nose down hard to reduce power is an 
uncomfortable means of control inputs 
“Conventional control is more responsive 

“I don't like holding the input (for V-CAS) as opposed to adding an input then neutralizing 
controls. ” ; 

“Less control input seemed to be required when using conventional controls 
“I think I was more comfortable with the conventional, the VCAS was easy to fly but I felt I 
did a better job of anticipating the power changes necessary. That being said I seemed to 
track the guidance better while flying the VCAS 

Comments given by pilots who preferred the V-CAS mainly relate to workload 
reduction. Comments supportive of V-CAS included the following: 

“(V-CAS) reduced my workload by keeping the aircraft on speed and coordinated 
“The auto coordination was a tremendous help ” 


“VCAS increased the precision 
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Participant’s preference towards the guidance symbology and terrain portrayal within 
different control types was assessed using block questionnaires. These findings are presented 
in Table 4. 


Pilot Rating 

Control Type 

Preference of Terrain 

Preference of Guidance Svmbologv I 

BSBG 

EBG 

PRFD 

HITS 

IFR 

Conventional 

0 

4 

3 

0 

V-CAS 

0 

4 

3 

0 

Non-IFR 

Conventional 

0 

8 

3 

5 

V-CAS 

0 

8 

3 

5 


Table 4 - Summary of Subjective Preference on Terrain and Guidance Symbology 


IFR rated participants had a strong preference (100%) towards EBG and PRFD 
regardless of control type. For non-IFR participants, EBG is strongly preferred (100%); 
however, there is a split in preference of guidance symbology, about 38% of participants 
prefer the PRFD and the rest (62%) prefer HITS. These results are consistent with the SA 
results, in which EBG has a significantly higher score than BSBG across all pilot ratings. 
However, the difference in workload scores for these two terrain types was not significant. 
Finally, as all evaluation pilots had a subjective preference for the EBG terrain portrayal 
concept, hypothesis 5 is accepted. 

Discussions and Conclusions 

This study aimed to investigate, via flight evaluations, the effect of SVS terrain 
portrayal and guidance symbology and V-CAS control on pilot performance and perceived 
workload and situational awareness during the conduct of advanced approach procedures. 
From the preceding analyses, the following major findings are presented: 

Flight technical performance 

Surprisingly, pilot rating (IFR versus non-IFR) was not a statistically significant factor 
for the FTP metrics used this study. These FTP metrics included horizontal deviation RMSE, 
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vertical deviation RMSE, airspeed RMSE, and proportion of time within standards. Of 
particular interest in this study is the comparison of the advanced concepts relative to their 
baseline counterparts in regards to FTP. Comparing V-CAS versus baseline flight control; 
EBG versus BSBG terrain portrayal; and HITS versus PRFD, no significant differences were 
found except for the effect of guidance symbology on vertical path deviation. For vertical 
tracking, HITS resulted in less deviation than the PRFD across all other conditions. Two 
significant interaction effects were found relative to lateral path deviation. These 
interactions involved pilot rating and control type; and pilot rating, control type, and guidance 
symbology. For lateral tracking, V-CAS reduced the deviation of non-IFR pilots; while for 
IFR pilots, V-CAS had a negative effect. Considering the percentage of time the subjects 
maintained flight technical performance within the specified standards, the data showed no 
significant effects. In summary, based on the FTP results hypothesis 1 is accepted for 
vertical tracking but rejected for lateral tracking and airspace regulation. Hypothesis 2 and 3 
cannot be accepted. We can accept hypothesis 4 which states that the terrain portrayal 
concept will not affect flight technical performance. 

Workload and Situational Awareness 

The subject’s perceptions of workload did not differ significantly between the 
treatments. For situation awareness, the only significant factor found was the terrain 
portrayal, with the EBG concept having higher perceived SA than the BSBG concept. This 
result is not difficult to understand since EBG provides relatively detailed terrain information. 
It should also be recognized that this improved awareness was obtained without negatively 


impacting workload or FTP. 
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Factors Influencing Results 

In this experiment, with the exception of improved perceived SA from the EBG terrain 
portrayal, the different concepts yielded only minor changes in FTP, workload, and SA. 

There are several important factors that probably influenced these results and these factors 
should be considered before applying the results beyond the context of the experiment. 

The first factor is the effect of prior pilot experience and training. In general, the pilots 
had much more experience with the conventional or baseline concepts prior to the experiment. 
For example, all the pilots’ previous training and operations would have been conducted 
using conventional control systems. While the subjects were given training and time to 
practice with the concepts until they demonstrated adequate proficiency and felt prepared to 
perform the evaluations, it is unlikely that they had reached maximal performance with the 
advanced concepts. In the case of the V-CAS, it is clear that significant negative transfer 
from previous experience and training was a factor. As mentioned previously, the original 
design user group for this system was ab-initio pilots and certain design features that would 
benefit this group such as lateral stick inputs commanding turn-rate (i.e. bank angle) rather 
than roll-rate as in a conventional aircraft were found to be distracting by a number of the 
evaluation pilots with their relatively brief exposure to the system. Had this system been 
designed for pilots already trained on conventional control systems, a different set of 
command responses may have minimized transition issues while retaining many of the 
benefits of the underlying technology. 

A second factor is related to flight task itself. In this study, evaluation pilots were able 
to allocate their full attention to the task of flying the approach. In the context of typical GA 
operations and perhaps more so in future operations in which some traffic separation 
responsibilities may be delegated to the cockpit, pilots are required to share their attention 
between multiple tasks. It is possible that the isolated approach task used in this study was 
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not well suited to revealing the potential operational benefits of the advanced concepts. 
Evaluation scenarios in which the pilot must perform other tasks, in addition to flying the 
airplane, could well uncover significant differences not found in this study. 

A final factor to consider is the implementation details of the systems used in this 
experiment versus the more general concepts. The advanced display and control concepts are 
the result of many low-level design and implementation details. A minor, easily remedied 
deficiency, in any of these details can color the evaluation of the entire concept. For example, 
strong winds aloft during some of the flights uncovered a previously unseen interaction 
between the V-CAS and the steep approach segment. At full-forward stick, the V-CAS 
commands a descent angle of 7 degrees relative to the air mass data from which the actual 
descent angle is derived. Pre-experiment trails had shown this limit to be adequate to track 
the 6 degree descent angle of the steep approach segment. Strong tail winds during some of 
the evaluation flights resulted in an inertial descent angle at full stick that was equal to or less 
than the 6-degree angle of the steep approach segment. Needless to say, pilots that could not 
follow the segment precisely because of this limit found the experience frustrating and these 
encounters affected both flight technical and subjective performance measures. A more 
mature or refined V-CAS design could maintain simplified control through a greater expanse 
of the physical flight envelope of the aircraft and may elicit different pilot reaction than seen 
in this study. 

With these thoughts in mind, the results of this study should be seen as a contribution 
to the growing body of experience with advanced control and SVS display concepts. The 
study is relatively unique in that it provides an initial investigation into potential interactions 
between display and control concepts in a flight test environment. With the exception of the 
enhanced perceived awareness afforded by the EBG terrain portrayal relative to the baseline, 
the effects seen in this study were modest and in most cases, no significant differences 
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between conditions were observed. These results suggest several general observations. First, 
while the advanced guidance and control concepts may have the potential to improve flight 
performance, this improvement is likely to be limited when used by highly trained, proficient 
pilots with full attention dedicated to controlling the aircraft. Even the non-IFR rated pilots 
participating in this study could perform the challenging approach task with reasonable 
accuracy using the conventional, state-of-art displays and controls. That said, it should be 
recognized that the “conventional, state of the art displays” used in this study included a large 
format primary flight display with a flight director and also a separate, large format 
navigation display. While this combination has become the norm on many newly 
manufactured small aircraft, it is far beyond the “steam-gauge”(electro-mechanical round 
dials) panels found in the majority of the operational fleet. Also, while performance 
improvements are likely to be modest for fully attentive pilots, gains for pilots having to 
divide their attention with other responsibilities may be much more meaningful, particular in 
terms of error prevention, detection, and recovery. Future evaluations should include 
operationally representative scenarios requiring the evaluation pilots to divide their attention 
between the control task and other cockpit responsibilities. 

Another observation is that achieving potential performance improvements depends on 
many details of the implementation. A minor deficiency in any of these details or their 
interactions may overwhelm the potential benefit of the integrated concept. The final 
observation is that transitioning concepts from simulation to flight or even expanding the 
flight envelope of “flight proven” concepts is likely to uncover previously unseen or 
unknown deficiencies, despite rigorous simulation and build-up. The flight evaluation 
schedule should provide sufficient time to thoroughly screen, and if necessary refine both 
technical concepts and experimental procedures prior to data collection runs. Even then, 
novel factors are frequently encountered during the formal flight evaluation process. While 
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these encounters may invalidate pre-flight expectations, they also afford the learning 
opportunities from which new knowledge and progress flow. 
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